Assurance  Cases  for  Design 
Analysis  of  Complex  System 
of  Systems  Software 


Presented  at 

AIAA  I nfotech@ Aerospace  Conference 
Software  Assurance  Session 
8  April  2009 

Stephen  Blanchette,  Jr. 


Software  Engineering  Institute 


Carnegie  Mellon 


©2009  Carnegie  Mellon  University 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

08  APR  2009 


2.  REPORT  TYPE 


3.  DATES  COVERED 

00-00-2009  to  00-00-2009 


4.  TITLE  AND  SUBTITLE 

Assurance  Cases  for  Design  Analysis  of  Complex  System  of  Systems 
Software 

6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Carnegie  Mellon  University, Software  Engineering 
Institute, Pittsburgh, PA, 15213 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

presented  at  the  American  Institute  for  Aeronautics  and  Astronautics  (AIAA)  Inf otech@  Aerospace 
Conference,  7-9  Apr  2009,  Seattle,  WA. 


14.  ABSTRACT 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 


a.  REPORT 

unclassified 


b.  ABSTRACT 

unclassified 


c.  THIS  PAGE 

unclassified 


17.  LIMITATION  OF 

18.  NUMBER 

ABSTRACT 

OF  PAGES 

Same  as 

16 

Report  (SAR) 

19a.  NAME  OF 
RESPONSIBLE  PERSON 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Problem:  SoS  are  Common,  but  Difficult 


Huge  systems  of  systems  (SoS)  development  efforts 

•  Have  become  quite  commonplace  as  aerospace/defense  solutions 

-  The  Global  Earth  Observation  System  of  Systems  (GEOSS) 

-  The  US  Army’s  Future  Combat  Systems 

-  The  US  Coast  Guard’s  Deepwater 

-  Etc. 

•  Exhibit  extreme  complexity  -  they're  hard  to  understand  in  detail 

•  Are  difficult,  even  impossible,  to  test  adequately  using  traditional  methods 

•  Use  software  as  an  enabler  of  SoS  functionality 

Often,  SoS  elements  are  dispersed  geographically,  adding  complexity  and  making 
predictability  of  behavior  difficult 

How  can  project  management,  stakeholders,  and  decision  makers  achieve  some 
reasonable  level  of  confidence  that  software  for  a  large-scale,  dispersed  SoS  will 
meet  operational  needs,  even  before  development  of  much  of  the  software? 
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Recent  Experience  Led  to  Assurance  Case  Use 


An  SEI  team  was  tasked  with  answering  this  question  in  the  face  of: 

•  the  software  design  was  actually  many  software  designs  documented  in  many 
places  and  with  a  tremendous  volume  of  data 

•  the  designs  were  not  yet  complete  and  would  not  be  complete  until  after  many 
production  decisions  had  to  be  made 

•  the  need  to  relate  the  software  to  operational  needs 

•  the  desire  to  base  conclusions  as  much  as  possible  on  actual  data  rather  than  on 
optimistic  plans  and  confident  assertions 

Constraints 

•  short  time  frame:  a  project  level  review  had  a  hard  deadline 

•  limited  availability  of  personnel,  all  with  varying  levels  of  domain  knowledge 
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We  Had  to  Relate  Operational  Needs  to 
Software 

The  top-level  SoS  requirements  were  expressed  in  military  terms. 

•  A  requirements  analysis  would  be  repeating  work  already  done 

•  Errors  in  requirements  traceability  could  skew  the  entire  analysis 

We  decided  against  re-casting  the  operational  needs  in  terms  of  software  and 
instead  analyzed  the  software  contributions  to  the  definitive  characterization  of 
those  needs — the  Key  Performance  Parameters  (KPPs). 

The  size  of  the  analysis  space,  the  complexity  of  the  task,  and  the  desire  to 
leverage  data  suggested  an  assurance  case  approach 

•  The  analysis  was  a  structured  decomposition  of  each  KPP  into  more  precise 
statements  that  could  be  more  readily  assessed  in  terms  of  evidence. 

-  Challenge:  be  logical  and  consistent  but  avoid  accumulating  too  much  detail 

-  Use  engineering  judgment  to  leap  from  higher-level  concepts  to  lower-level  ones 
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An  Assurance  Case  is  a  Structured  Argument 


An  assurance  case 


•  is  a  generalization  of  a  safety  case 


•  presents  an  argument  (similar  to  a  legal  case)  that  a  system 
has  or  satisfies  some  property  in  a  given  context 


•  requires  claims,  evidence,  and  an  argument  linking  evidence 
to  claims 

•  should  be  sound  and  complete  to  justify  belief  in  the  main 
claim 

•  should  be  based  on  objective  evidence 


Goal  Structuring  Notation  (GSN) 

•  graphically  presents  the  argument  by  showing  how  claims  are 
broken  down  into  sub-claims  until  arriving  at  a  sub-claim 
supported  by  evidence. 

For  our  purposes,  we  used  assurance  cases  to 
demonstrate  that  a  SoS  software  design  supported  each 
of  the  SoS  KPPs,  as  in  the  claim  “The  SoS  will  satisfy 
KPP  k” 
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An  Example 


Assume  we’re  developing  a  SoS  for  a  DoD  project 

•  The  SoS  must  exchange  information  with  other  systems 

-Our  SoS  is  subject  to  the  Net  Ready  Key  Performance 
Parameter  (NR  KPP): 

•The  system. ..must  support  Net-Centric  military 
operations.  The... system... must  be  able  to  enter  and 
be  managed  in  the  network,  and  exchange  data  in  a 
secure  manner  to  enhance  mission  effectiveness. 

The. . .system.. .must  continuously  provide  survivable, 
interoperable,  secure,  and  operationally  effective 

information  exchanges  to  enable  a  Net-Centric 
military  capability. 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction,  “Interoperability  and  Supportability  of  Information 

Technology  and  National  Security  Systems,”  CJCSI  621 2.01  E,  2008. 
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We  Can  Express  the  NR  KPP  Diagrammatically 


A1 

ft  is  assumed  that 
hardware  components 
of  the  network  can  be 
ignored  in  this 
analysis. 


Ntel 

This  diagram  focuses 
on  threshold  level  KPP 
satisfaction 
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Primary  Sub-claim  #6 


The  SoS  is  able  to 
exchange  data  in  a 
secure  manner  to 
enhance  mission 
effectiveness 


Clm7 

The  security 
requirements  (e.g.,  data 
security,  information 
assurance,  access 
control)  for  the  SoS  are 
met 
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Primary  Sub-claim  #13 


Clm13 

The  SoS  continuously 
provides  survivable, 
interoperable,  secure,  and 
operationally  effective 
information  exchanges 


Clml  1 

The  SoS  supports 
information  exchanges 
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The  SoS 
information 
exchanges  occur 
within  prescribed 
timelines 
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A  Diagram  Helps  Visualize  the  Completed  Case 
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The  SoS  is  able  to 
exchange  data  in  a 
secure  manner  to 
enhance  mission 
effectiveness 


Ntel 

This  diagram  focuses 
on  threshold  level  KPP 
satisfaction 
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The  SoS  continuously 
provides  survivable, 
interoperable,  secure,  and 
operationally  effective 
information  exchanges 
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features  is 
adequately  planned 
and  resourced 


Logic  Should  Hold. ..Even  Without  the  Diagram 
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Scoring  Can  be  Used  to  Express  Risk 

First,  Develop  Scoring  Rules 


For  Evidence 

Green 

Evidence  is  complete  and  adequate 

Yellow 

Evidence  is  incomplete  or  planned  for  the  future 

Red 

Evidence  is  complete  but  inadequate,  planned  but 
now  late,  or  non-existent 

For  Claims 

Green 

All  lower-level  claims  and  supporting  evidence 
are  green 

Yellow 

Some  lower-level  claims  and  supporting  evidence 
are  a  combination  of  yellow  and  red 

Red 

All,  or  an  overwhelming  majority  of,  lower-level 
claims  and  supporting  evidence  are  red 

Then,  Work  Upward  from  Evidence.. 
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Scored  Diagram  Provides  a  Roadmap... 


Quality  of  the  Evidence  Drives  Assessment  of  Claims... and  Relative  Risk 
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Assurance  Cases  are  Helpful  in  the  SoS  Space 


Assurance  cases  gave  us  a  way  of  organizing  a  nebulous  task  and  gave 
us  a  means  of  selecting  among  innumerable  artifacts  to  study.  They 
brought  order  to  complexity. 

•  Due  to  time  constraints  we  had  to  focus  on  big  picture  risks 

•  A  more  thorough  analysis  might  have  identified  additional  risks  or  strong 
points 

The  assurance  case  technique  is  a  powerful  tool  for  analyzing  large  and 
complex  SoS  software  design. 

•  It  provides  a  means  of  taking  a  crosscutting  look  at  SoS 

•  It  gives  managers  answers  about  design  progress  that  are  rooted  in  facts  and 
data  instead  of  opinions  based  upon  hope  and  best  intentions. 
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